What is claimed is: 

1. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information 

space; 

a display means for displaying an image on said operation surfaces; 
an imaging means for picking up an image of said operation surfaces; 
an environment type computer stationarily arranged in said information 
space; and 

one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface 
thereof; 

said environmental type computer being adapted to execute; 

a processing operation of recognizing the identification information and 
the position information of each of said physical objects in said visual marker; 

a processing operation of recognizing the digital object dropped to a site 
on the surface of each of said physical objects; and 

a processing operation of forming link information for linking the digital 
object to the dropped site on the surface for each of said physical objects. 

2. The information input/output system according to claim 1, wherein 
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at least one of said physical objects is a portable computer capable of being 
moved in said information space and exchanging digital objects with other 
computers. 

3. The information input/output system according to claim 1, wherein 
at least one of said physical objects is a portable computer capable of 

being moved in said information space and exchanging digital objects with 
other computers; and 

said environment type computer can expand a mouse/cursor operation on 
the installed portable computer onto said operation surfaces. 

4. The information input/output system according to claim 1, wherein 
said imaging means can identify the position indicated by an optical 

pointer for indicating a specific position by irradiating a beam of light with a 
predetermined wavelength; and 

the user is allowed to indicate a position in said information space by 
means of coordinates and the optical pointer. 

5. The information input/output system according to claim 1, wherein 
said environment type computer executes a processing operation of calling the 
linked digital object and/or displaying the digital object to the user in response 
to a user operation applied to the site of forming the link information on the 
surface of each of the physical objects on the basis of the image picked up by 
said imaging means. 
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6. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information 

space; 

a display means for displaying an image on said operation surfaces; 
an imaging means for picking up an image of said operation surfaces; 
an environment type computer stationarily arranged in said information 

space; 

one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface 
thereof; and 

a portable ID recognition device adapted to recognize/identify the visual 
marker on the surface of each of said physical objects. 

7. The information input/output system according to claim 6, wherein 
said environmental type computer is adapted to execute; 

a processing operation of receiving the identification information of the 
source object and that of the destination object from said ID recognition 
device; and 

a processing operation of applying an action specified on the basis of the 
combination of the type of the source object and that of the destination object. 



8. The information input/output system according to claim 6, wherein 
at least one of said physical objects is a portable computer capable of being 
moved in said information space and exchanging digital objects with other 
computers. 

9. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information 

space; 

a display means for displaying an image on said operation surfaces; 
an imaging means for picking up an image of said operation surfaces; 
an environment type computer stationarily arranged in said information 
space; and 

one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface 
thereof; 

at least one of said physical objects being a virtual camera having a 
virtual imaging direction as specified by the position information contained in 
the visual marker. 

10. The information input/output system according to claim 9, wherein 
said environmental type computer is adapted to execute; 
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a processing operation of recognizing/identifying said virtual camera and 
identifying the position information and the virtual imaging direction; 

a processing operation of generating a virtual picked up image according 
to the position information and the virtual imaging direction; and 

a processing operation of displaying the virtual picked up image to the 

user. 

1 1 . The information input/output system according to claim 9, wherein 
at least one of said physical objects is a portable computer that can move in 
said information space and exchange digital objects with another computer. 

12. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information 

space; 

a display means for displaying an image on said operation surfaces; 
an imaging means for picking up an image of said operation surfaces; 
an environment type computer stationarily arranged in said information 

space; 

one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface 
thereof; and 
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an optical pointer adapted to point a specific position by irradiating a 
beam of light with a predetermined wavelength; 

said imaging means being capable of identifying the position pointed by 
said optical pointer. 

13. The information input/output system according to claim 12, wherein 
said environment type computer executes on the basis of the image 

picked up by said imaging means; 

a processing operation of recognizing the user operation on said 
operation surfaces, using said optical pointer; and 

a processing operation of controlling the display of the digital object by 
said display means according to the result of the recognition. 

14. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information 

space; 

a display means for displaying an image on said operation surfaces; 
an imaging means for picking up an image of said operation surfaces; 
an environment type computer stationarily arranged in said information 
space; and 

one or more than one portable computers capable of exchanging digital 
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objects with other computers; 

said portable computers being capable of retrieving a three-dimensional 
model and/or generating a three-dimensional model thereon; 

said environment type computer being adapted to execute; 

a processing operation of generating an image to be projected on said 
operation surfaces on the basis of the information for drawing a three- 
dimensional model in response to the application of the user operation of 
taking out the three-dimensional model from one of said portable computers 
to said operation surfaces, and a processing operation of controlling a display 
of the projected image generated by said display means. 

15. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information 

space; 

a display means for displaying an image on said operation surfaces; 
an imaging means for picking up an image of said operation surfaces; 
an environment type computer stationarily arranged in said information 

space; 

one or more than one portable computers capable of exchanging digital 
objects with other computers; 
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one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface 
thereof; and 

a portable ID recognition device adapted to recognize the visual marker 
on the surface of each of said physical objects and being capable of holding or 
releasing the recognized identification information. 

16. The information input/output system according to claim 15, wherein 
said environment type computer executes; 

a processing operation of obtaining a digital object corresponding to the 
held identification information in response to the holding operation of said ID 
recognition device; and 

a processing operation of transferring a digital object corresponding to 
the held identification information to a nearby physical object in response to 
the releasing operation of said ID recognition device. 

17. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information 

space; 

a display means for displaying an image on said operation surfaces; 
an imaging means for picking up an image of said operation surfaces; 
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an environment type computer stationarily arranged in said information 

space; 

one or more than one portable computers capable of exchanging digital 
objects with other computers; and 

one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface 
thereof; 

said environment type computer being adapted to allow mouse/cursor 
operations on said portable computers to extend to said operation surfaces and 
display a rubber band as visual feedback according to the area on said 
operation surfaces as defined by means of said mouse/cursor. 

18. The information input/output system according to claim 17, wherein 
said environment type computer is adapted to pick up an image of the area 
defined by the rubber band by means of said imaging means and take the 
picked up image in said information space as digital object. 

19. The information input/output system according to claim 17, wherein 
another object can apply a process or a method it possesses to the rubber- 
banded original digital object in response to an action of said original object 
of being dropped on said another object. 

20. The information input/output system according to claim 19, wherein 
said another object is a physical object possessing a process or a method for 
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printing a document. 

2 1 . The information input/output system according to claim 19, wherein 
said another object is a name card possessing a process or a method for mailing 
a document to the mail address corresponding to the name on it. 

22. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said information space having one or 
more than one operation surfaces arranged in said information space, a display 
means for displaying an image on said operation surfaces, an imaging means 
for picking up an image of said operation surfaces, an environment type 
computer stationarily arranged in said information space and one or more than 
one physical objects mounted on said operation surfaces and containing a 
visually identifiable visual marker on the surface thereof, said system 
comprising: 

a step of firstly recognizing the identification information and the 
position information of each of said physical objects from the visual marker; 

a step of secondly recognizing the digital object dropped to a site on the 
surface of each of said physical objects; and 

a step of forming link information for linking the digital object to the 
dropped site on the surface for each of said physical objects. 

23. The information input/output system according to claim 22, wherein 
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at least one of said physical objects is a portable computer capable of 
being moved in said information space and exchanging digital objects with 
other computers; and 

said environment type computer can expand a mouse/cursor operation on 
the installed portable computer onto said operation surfaces in said second 
recognition step. 

24. The information input/output system according to claim 22, wherein 
said imaging means can identify the position indicated by an optical 

pointer for indicating a specific position by irradiating a beam of light with a 
predetermined wavelength; and 

the user is allowed to indicate a position in said information space by 
means of coordinates and the optical pointer. 

25. The information input/output system according to claim 22, further 
comprising: 

a step of calling the linked digital object and/or displaying the digital 
object to the user in response to a user operation applied to the site of forming 
the link information on the surface of each of the physical objects. 

26. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system having one or more than 
one operation surfaces arranged in said information space, a display means for 
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displaying an image on said operation surfaces, an imaging means for picking 
up an image of said operation surfaces, an environment type computer 
stationarily arranged in said information space, one or more than one physical 
objects mounted on said operation surfaces and containing a visually 
identifiable visual marker on the surface thereof and a portable ID recognition 
device adapted to recognize/identify the visual marker on the surface of each 
of said physical objects, said system comprising: 

a step of receiving the identification information of the source object and 
that of the destination object from said ID recognition device; and 

a step of applying an action specified on the basis of the combination of 
the type of the source object and that of the destination object. 

27. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
Space of a computer into the real world, said system having one or more than 
one operation surfaces arranged in said information space, a display means for 
displaying an image on said operation surfaces, an imaging means for picking 
up an image of said operation surfaces, an environment type computer 
stationarily arranged in said information space and one or more than one 
physical objects mounted on said operation surfaces and containing a visually 
identifiable visual marker on the surface thereof, at least one of said physical 
objects being a virtual camera having a virtual imaging direction as specified 
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by the position information contained in the visual marker, said system 
comprising: 

a step of recognizing/identifying said virtual camera and identifying the 
position information and the virtual imaging direction; 

a step of generating a virtual picked up image according to the position 
information and the virtual imaging direction; and 

a step of displaying the virtual picked up image to the user. 

28. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system having one or more than 
one operation surfaces arranged in said information space, a display means for 
displaying an image on said operation surfaces, an imaging means for picking 
up an image of said operation surfaces, an environment type computer 
stationarily arranged in said information space, one or more than one physical 
objects mounted on said operation surfaces and containing a visually 
identifiable visual marker on the surface thereof and an optical pointer adapted 
to point a specific position by irradiating a beam of light with a predetermined 
wavelength, said imaging means being capable of identifying the position 
pointed by said optical pointer, said system comprising: 

a step of recognizing the user operation on said operation surfaces, using 
said optical pointer; and 
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a step of controlling the display of the digital object by said display 
means according to the result of the recognition. 

29. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system having one or more than 
one operation surfaces arranged in said information space, a display means for 
displaying an image on said operation surfaces, an imaging means for picking 
up an image of said operation surfaces, an environment type computer 
stationarily arranged in said information space and one or more than one 
portable computers capable of exchanging digital objects with other computers, 
said portable computers being capable of retrieving a three-dimensional model 
and/or generating a three-dimensional model thereon, said system comprising: 

a step of generating an image to be projected on said operation surfaces 
on the basis of the information for drawing a three-dimensional model in 
response to the application of the user operation of taking out the three- 
dimensional model from one of said portable computers to said operation 
surfaces; and 

a step of controlling the display of said generated and projected image 
of said display means. 

30. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
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space of a computer into the real world, said system having one or more than 
one operation surfaces arranged in said information space, a display means for 
displaying an image on said operation surfaces, an imaging means for picking 
up an image of said operation surfaces, an environment type computer 
stationarily arranged in said information space and one or more than one 
portable computers capable of exchanging digital objects with other computers, 
one or more than one physical objects mounted on said operation surfaces and 
containing a visually identifiable visual marker on the surface thereof and a 
portable ID recognition device adapted to recognize the visual marker on the 
surface of each of said physical objects and being capable of holding or 
releasing the recognized identification information, said system comprising: 

a step of obtaining a digital object corresponding to the held 
identification information in response to the holding operation of said ID 
recognition device; and 

a step of transferring a digital object corresponding to the held 
identification information to a nearby physical object in response to the 
releasing operation of said ID recognition device. 

31. An information input/output system to be used for user operations 
relating to an object in an information space realized by expanding a digital 
space of a computer into the real world, said system having one or more than 
one operation surfaces arranged in said information space, a display means for 
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displaying an image on said operation surfaces, an imaging means for picking 
up an image of said operation surfaces, an environment type computer 
stationarily arranged in said information space, one or more than one portable 
computers capable of exchanging digital objects with other computers and one 
or more than one physical objects mounted on said operation surfaces and 
containing a visually identifiable visual marker on the surface thereof, said 
system comprising: 

a step of allowing mouse/cursor operations on said portable computers 
to extend to said operation surfaces; and 

a step of displaying a rubber band as visual feedback according to the 
scope on said operation surfaces as defined by means of said mouse/cursor. 

32. The information input/output system according to claim 31, further 
comprising: 

a step of picking up an image of the scope defined by the rubber band by 
means of said imaging means and taking the picked up image in said 
information space as digital object. 

33. The information input/output system according to claim 31, further 
comprising: 

a step of applying a process or a method possessed by said another 
object to the rubber-banded original digital object in response to an action of 
said original object of being dropped on said another object. 
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34. The information input/output system according to claim 33, wherein 
said another object is a physical object possessing a process or a method for 
printing a docmnent. 

35. The information input/output system according to claim 33, wherein 
said another object is name card possessing a process or a method for mailing 
a document to the mail address corresponding to the name on it. 
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